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(54) IIUIAGING DEVICE AND METHOD 



(57) A composite image having a natural appear- 
ance is obtained when a synthesis target image is com- 
bined with a background image. By photographing a 
subject under different photographic conditions, a plu- 
rality of synthesis target images are obtained and the 
resulting image data is stored beforehand in a synthesis 
target image memory. A background image is acquired 
by photography and a suitable synthesis target image to 
be combined with the bacl<ground image is selected by 
a circuit which searches synthesis target images. A 



color correction is applied to the selected synthesis tar- 
get image by an image processing circuit and the syn- 
thesis target Image following the color correction Is 
combined with the background Image. Since a synthe- 
sis target image suitable for combination with a back- 
ground image is selected and then is subjected to a 
color correction and image synthesis processing, a 
composite image having a natural appearance is 
obtained. 
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Description 

Technical Field 

5 [0001 ] This invention relates to an image sensing apparatus and method for outputting composite image data rep- 
resenting a composite image. 

Bacltqround Art 

10 [0002] A synthesis apparatus for combining an image of scenery and an image of a person obtained by photogra- 
phy has already been realized. In an apparatus of this kind, the person is photographed in advance and the image of 
the person is extracted from the Image obtained. The scenery is photographed separately from the person to obtain the 
scenery Image. The extracted image of the person is combined with the image of the scenery (image data of the portion 
of the scenery image where the image of the person is to be placed is replaced by the image data representing the 

75 image of he person). 

[0003] The photographic conditions that prevail when the image of the person is obtained (f rontlighting or backlight- 
ing, indoor or outdoor, strobe photography or not, etc.) often differ from the photographic conditions which prevail when 
the image of scenery is obtained. If these photographic conditions differ, the composite image may appear unnatural. 

20 Disclosure of the Invention 

[0004] An object of the present invention Is to obtain a composite image having a natural appearance. 

[0005] An image sensing apparatus according to a first aspect of the present invention is characterized by having 

synthesis target Image data storage means for storing data representing a plurality of synthesis target images obtained 

25 by photography under different photographic conditions; image sensing means for photographing bacicground and out- 
putting data representing a background image; synthesis target image selection means for selecting one Item of syn- 
thesis target image data suited to a background image from a plurality of items of synthesis target image data, which 
have been stored In the synthesis target Image data storage means, based upon photographic conditions prevailing at 
time of photography of the background image by the image sensing means; and image synthesis means for combining 

30 the synthesis target image selected by the synthesis target image selection means with the background Image output 
from the image sensing means, and outputting composite image data representing a composite image. 
[0006] The first aspect of the present invention provides also a method suited to the apparatus described above. 
Specifically, the method is characterized by storing data representing a plurality of synthesis target Images obtained by 
photography under different photographic conditions; photographing background and outputting data representing a 

35 background image; selecting one item of synthesis target image data suited to a background image from a stored plu- 
rality of items of synthesis target image data based upon photographic conditions prevailing at time of photography of 
the background image; and combining the selected synthesis target image with background image data and outputting 
composite image data representing a composite image. 

[0007] In accordance with the first aspect of the present invention, a subject which includes a synthesis target is 
40 photographed a plurality of times under different photographic conditions (photography outdoors with f rontlighting, pho- 
tography outdoors with backlighting, photography indoors, strobe photography indoors, etc.). The synthesis target 
images are extracted from tiiese plurality of subject images obtained by the photography performed the plurality of 
times. The data representing the synthesis target images is stored. 

[0008] Background is photographed and background image data representing the background image is obtained. 

45 [0009] On the basis of the photographic conditions that prevailed when the background was photographed (photo- 
graphic circumstances as to whether the background was photographed with backlighting orfrontlighting, color temper- 
ature, luminance of the background Image, etc.), synthesis target image data representing a synthesis target image the 
image characteristics whereof are near those of the background Image (near in terms of brightness and near in terms 
of color temperature, l.e., the photographic conditions of whteh are the same as or almost the same as those of the 

so background image) is selected (automatically or by having the user enter a selection command and then making the 
selection based upon the entered selection command) from the plural items of synthesis target image data that have 
been stored. 

[001 0] The synthesis target image represented by the synthesis target image data that has been selected and the 
background image are combined. 
55 [0011] Since a synthesis target image whose image characteristics are near those of the background image Is 
selected and combined with the background image, a composite image having a natural appearance is obtained. 
[0012] Preferably, photographic conditions at the time of photography of the synthesis target image are stored 
beforehand and, on the basis of photographic conditions corresponding to tiie synthesis target image data that has 
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been selected and photographic conditions of the background image, image adjustment processing for at least one of 
a color adjustment and a luminance adjustment is applied to the synthesis target image represented by the synthesis 
target image data that has been selected, and the synthesis target image data that has been subjected to the image 
adjustment processing is combined with the background image data. 
5 [001 3] The synthesis target image that has been selected is subjected to image adjustment processing for at least 
one of a color adjustment and a luminance adjustment. As a result, the image characteristics of the synthesis target 
image that has been subjected to the image adjustment processing will be nearer those of the background image. The 
composite Image obtained therefore appears more natural. 

[001 4] Furthermore, an image adjustment command for at least one of a color adjustment and a luminance adjust- 
Jft ment is allowed to be entered and, on the basis of the entered image adjustment command, image adjustment process- 
ing for at least one of a color adjustment and a luminance adjustment is applied to the synthesis target image 
represented by the synthesis target Image data that has been selected, and the synthesis target Image data that has 
been subjected to the image adjustment processing is combined with the background image data. 
[0015] A desired Image adjustment (color adjustment, luminance adjustment) can be performed by entering the 
IS image adjustment command. This makes it possible to perfomn image adjustment so as to obtain a composite image 
prefen'ed by the user. 

[0016] Preferably, a display unit for displaying Images is provided, the background image that has been sensed is 
displayed on the display unit and so is the composite image. 

[0017] Preferably, when the composite image is being displayed on the dlspl^ unit, an adjustment command for at 
20 least one of position and size of the synthesis target image being displayed on the display unit is allowed to be entered 
and an adjustment of at least one of a position and a size adjustment can be applied to the synthesis target image. 
[0018] The position of the synthesis target image and the size of the synthesis target image data can be adjusted 
while the synthesis target image being displayed on the display unit is observed. The synthesis target image can be 
adjusted to the appropriate position and size. 
2S [001 9] An image sensing apparatus according to a second aspect of the present invention is characterized by hav- 
ing synthesis target information storage means for storing data representing a synthesis target image and photographic 
conditions prevailing at time of photography of the synthesis target Image; image sensing means for photographing 
background and outputting data representing a background image; image adjustment processing means for applying 
image adjustment processing for at least one of a color adjustment and a luminance adjustment to the synthesis target 
30 image represented by the synthesis target image data, which has been stored in the synthesis target informaflon stor- 
age means, based upon the photographic conditions of the synthesis target image that have been stored in the synthe- 
sis target information storage means and photographic conditions at time of photography of the background by the 
Image sensing means; and image synthesis means for combining the synthesis target image data that has been sub- 
jected to the Image adjustment processing by the image adjustment processing means with the background image data 
35 output from the image sensing means, and outputting composite image data representing a composite image. 

[0020] The second aspect of the present invention provides also a method suited to the apparatus described 
above. Specifically, the method is characterized by storing data representing a synthesis target image and photographk: 
conditions prevailing at time of photography of the synthesis target image; obtaining data representing a background 
image by photographing background; applying image adjustment processing for at least one of a color adjustment and 
40 a luminance adjustment to a synthesis target image represented by synthesis target image data based upon the pho- 
tographic conditions of the synthesis target image and photographic conditions prevailing at time of photography of the 
background; and combining the synthesis target image data that has been subjected to the image adjustment process- 
ing with the background image data, and outputting composite image data representing a composite image. 
[0021] In accordance with the second aspect of the present invention, synthesis target image data and photo- 
ns graphic conditions of a synthesis target image are stored. In addition, background is photographed and background 
Image data is obtained. 

[0022] On the basis of the photographic conditions of the synthesis target image and the photographic conditions 
of the background image, image adjustment processing for at least one of a color adjustment and a luminance adjust- 
ment is applied to the synthesis target image represented by the synthesis target image data that has been stored, the 
50 synthesis target image data that has been subjected to the image adjustment processing is combined with the back- 
ground Image data and composite image data representing a composite image is output. 

[0023] Image adjustment processing for at least one of a color adjustment and a luminance adjustment of the syn- 
thesis target image is executed based upon the photographic conditions of the synthesis target image and background 
image. Image characteristics of the synthesis target image that has been subjected to the image adjustment processing 
55 are near those of the background image. As a result, the composite image obtained has a natural appearance. 
[0024] Preferably, synthesis target image data obtained by strobe photography is stored beforehand. 
[0025] Since a synthesis target image obtained by strobe photography is bright and sharp, image adjustments such 
as a color adjustment and luminance adjustment are comparatively easy to cany out. A natural composite image is 
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obtained through comparatively simple processing. 

[0026] An image adjustment command of at least one of a color adjustment and a luminance adjustment may be 
entered, image adjustment processing for at least one of a color adjustment and a luminance adjustment may be 

applied to the synthesis target image on the basis of the entered image adjustment command, and the synthesis target 
5 image data that has been subjected to the image adjustment processing may be combined with the bacl^ground image 
data. 

[0027] An image adjustment desired by the user is made possible by entering an image adjustment command for 
canying out an image adjustment desired by the user. A composite image that satisfies the user can thus be obtained. 
[0028] Further, a display unit for displaying images may be provided, the background image that has been photo- 

10 graphed may be displayed on the display unit and so may the composite image. 

[0029] Preferably, when the bacl<ground image and composite image are being displayed on the display unit, an 
adjustment command for at least one of position and size of the synthesis target image being displayed on the displ^ 
unit is entered and, on the basis of the entered adjustment command, an adjustment of at least one of position and size 
is applied to the synthesis target Image. 

15 [0030] The position and size of the synthesis target image can be adjusted while the synthesis target image being 
displayed on the display unit is observed. The synthesis target image can be adjusted to the appropriate position and 
size. 

Brief Description of the Drawings 

20 

[0031] 

Fig. 1 is a blocl< diagram illustrating the electrical construction of an image sensing apparatus; 

Fig. 2 illustrates the structure of a memory file of a synthesis target image; 
25 Fig. 3 Illustrates an example of a synthesis target image under strobe photography; 

Fig. 4 Illustrates an example of a synthesis target image under front-lit photography; 

Fig. 5 illustrates an example of a synthesis target image under bacl<-lit photography in cloudy weather; 

Fig. 6 illustrates the structure of a memory file of a bacl<ground image; 

Fig. 7 illustrates an example of a bacl<ground image; 
30 Fig. 8 Illustrates an example of a composite image; 

Fig. 9 is a flowchart Illustrating the procedure of image synthesis processing; 

Fig. 1 0 is a flowchart Illustrating the procedure of image synthesis processing; 

Fig. 11 is a flowchart illustrating the procedure of image synthesis processing; 

Fig. 12 is a flowchart illustrating the procedure of image synthesis processing; 
35 Fig. 13 illustrates an example of a screen for manual setting of color balance; 

Rgs. 1 4A to 1 4C illustrate the relationship between color-balance selection values and amounts of correction; 

Rg. 15 Illustrates an example of a screen for manual setting of brightness; 

Fig. 1 6 illustrates the relationship between brightness selection values and amounts of con-ectlon; 

Fig. 1 7 is a flowchart Illustrating the procedure of first color correction processing; 
40 Fig. 1 8 is a flowchart illustrating the procedure of second color correction processing; 

Fig. 1 9 illustrates the relationship between photographic circumstances and correction coefficients for color correc- 
tion; 

Rg. 20 Is a flowchart Illustrating the procedure of third color correction processing; 

Fig. 21 is a flowchart Illustrating the procedure of third color correction processing; 
45 Fig. 22 is a flowchart illustrating the procedure of image synthesis processing; 

Fig. 23 is a flowchart illustrating the procedure of image synthesis processing; 

Fig. 24 is a flowchart illustrating the procedure of image synthesis processing; 

Fig. 25 is a flowchart Illustrating the procedure of image synthesis processing; 

Rg. 26 Illustrates an example of a screen for manual setting of contrast; 
50 Fig. 27 illustrates an example of contrast correction curves; 

Fig. 28 is a flowchart illustrating the procedure of fourth color con'ection processing; and 

Fig. 29 is a flowchart Illustrating the procedure of fourth color con^ion processing. 
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Best Mode for Carrying Out the Invention 

Rrst Embodiment 

5 (1 ) Overview of image sensing apparatus 

[0032] Fig. 1 is a blocl( diagram lliustrating tlie electrical construction of an image sensing apparatus. 
[0033] Tills image sensing apparatus pfiotographs a subject (e.g., a person) in advance under different photo- 
graphic conditions, cuts out (extracts) a specific portion (e.g. , a portion corresponding to the person) from each of a plu- 

70 raiity of the images obtained by photography and stores synthesis target image data representing each of the plurality 
of extracted images (refen-ed to as "synthesis target images" below). The apparatus subsequently photographs a sub- 
ject that will constitute background. The apparatus combines one of the plurality of synthesis target images stored in 
advance with the background image obtained by photography (replaces image data of the portion of the background 
image where the synthesis target image is to be placed with image data of the synthesis target Image). For example, if 

75 the synthesis target image is the image of a person and the background image Is the image of scenery, then the appa- 
ratus photographs the person in advance under different photographic conditions and stores the results. The apparatus 
subsequently photographs the scenery, selects the image of the person photographed under photographic conditions 
that match those of the scenery image obtained by this photography and combines the selected image of the person 
with the scenery image. The details will become apparent from the description that follows. 

20 [0034] The image sensing apparatus Includes a synthesis target image memory 6. Synthesis target image data 
representing a plurality of synthesis target Images shot under different photographic conditions has been stored In the 
synthesis target image memory 6. 

[0035] The Image sensing apparatus includes a key switch group 20. The key switch group 20 includes a dial 21 for 
turning the power supply of the image sensing apparatus off and for setting a picture-taking mode, a picture-taking and 
2S synthesis mode, a synthesis mode, a synthesis target image selection mode and other modes (the power supply being 
turned on by setting the picture-taking mode, picture-taking and synthesis mode, synthesis mode, synthesis target 
image selection mode or other mode); an arrow key pad 22 for designating a position to which a synthesis target image 
displayed on an image-display liquid crystal display unit 14 is to be moved; a push switch 23 which includes minus and 
plus buttons for designating a reduction ratio and enlargement ratio of a synthesis target image displayed on the image- 
so display liquid crystal display unit 14; an execute button 24 for applying various execution commands such as a com- 
mand to start synthesis processing; and a manual-setting liquid crystal display unit 25 whtoh presents various displays 
for manually setting color balance, brightness and contrast. Signals output from these switches and the like are input to 
a synthesis target image search circuit 8, an image processing circuit 1 1 and a synthesis processing circuit 1 5. 
[0036] The picture-taking mode is a mode in which a subject (a person, scenery, or other subject) is photographed 
35 without performing image synthesis. The picture-taking and synthesis mode is a mode in which a subject that will serve 
as a background image is photographed and a synthesis target image is combined with the background image obtained 
by photography The synthesis mode is a mode in which a synthesis target Image is combined with a background image 
represented by background image data that has been stored in a background image memory 5. 
[0037] The light image of the background is formed on the photoreceptor surface of an image sensing device 2 
40 through an imaging lens 1 . A video signal representing the background image is output from the image sensing device 
2 and is converted to RGB digital image data in an analog/digital conversion circuit 3. The digital image data output from 
the analog/digital conversion circuit 3 enters a signal processing circuit 4. 

[0038] The image sensing apparatus includes an exposure control circuit 7. Image data output from the analog/dig- 
ital conversion circuit 3 is applied to the exposure control circuit 7. On the basis of the input image data, the exposure 

45 control circuit 7 performs exposure control such as an f-stop number adjustment and shutter-speed adjustment for pho- 
tography. Exposure control can also be earned out based upon an output signal from a metering device (sensor). On 
the basis of the entered image data, the exposure control circuit 7 calculates the average luminance of the background 
image and the spot luminance at the center of the background image. The exposure control circuit 7 further determines 
whether photography is backilt or frontlit photography based upon the luminance difference between average lumi- 

50 nance and spot luminance. 

[0039] The signal processing circuit 4 subjects image data to a gamma con-ection. Image data output from the sig- 
nal processing circuit 4 is input to the background image memory 5, where the data is stored temporarily 
[0040] The image sensing apparatus includes also the synthesis target image search circuit 8. The latter is a circuit 
which searches synthesis target images acquired under different photographic conditions for a synthesis target image 

55 that suits the captured background image. 

[0041] The Image sensing apparatus includes also a strobe unit 9. Data representing the fact that the strobe has 
been fired or whettier or not the strobe has been fired is input to the synthesis target image search circuit 8 from the 
strobe unit 9. 
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[0042] Synthesis target image data suited to the bacl<ground image found by the synthesis target image search cir- 
cuit 8 is read out of the syrrthesis target image memory 6 and Input to the image processing circuit 1 1 via the synthesis 
target image search circuit 8. Data representing the bacl<ground image that has been stored temporarily in the bacl<- 
ground image memory 5 aiso is input to the image processing circuit 1 1 . 
5 [0043] The image sensing apparatus further includes a color temperature sensor 1 0. Color temperature information 
(data) output from the color temperature sensor 1 0 is input to the image processing circuit 1 1 . The latter executes a first 
color correction, described later, based upon the entered color temperature infonnatlon. 

[0044] Baclcground image data obtained by photography is input to a digital/analog conversion circuit 13 via the 
image processing circuit 11 and synthesis processing circuit 15. The digital/analog conversion circuit 13 converts the 
10 background image data to an analog video signal. The analog video signal obtained by the conversion is applied to the 
image-display liquid crystal display unit 14 so that the background image is displayed. 

[0045] In image synthesis processing, synthesis target image data is applied to the synthesis processing circuit 1 5 
via the image processing circuit 11. The synthesis processing circuit 15 executes superposition synthesis in such a 
manner that the synthesis target image is superimposed on the background image. Data representing the image 
75 obtained by superposition synthesis Is supplied from the synthesis processing circuit 15 to the image-display liquid 
crystal display unit 14 via the digital/analog conversion circuit 13. The synthesis target Image superimposed on the 
background image is displayed on the image-display liquid crystal display unit 14. 

[0046] If a move command is applied to the synthesis processing circuit 1 5 from the arrow key pad 22, the synthesis 
processing circuit 15 applies move processing (a position-data offset, etc.) to the synthesis target image data in such a 

20 manner that the synthesis target image, whtoh Is being superimposed on the background image, will move (this repre- 
sents adjustment of the position of the synthesis target image). Further, if an enlarge or reduce command is applied to 
the synthesis processing circuit 15 from the push switch 23, the synthesis processing circuit 15 executes processing 
(pixel interpolation, etc.) to enlarge or processing (pixel downsampling, etc.) to reduce the synthesis target Image being 
superimposed on the background image (this represents adjustment of the size of the synthesis target image). 

25 [0047] If the execute button 24 is pressed by the user when a synthesis target image is being superimposed on the 
background image, the portion at which the synthesis target image is located is out from the background image. The 
synthesis target image is fit into the portion that has been cut away (this represents image synthesis). Composite Image 
data representing this composite image is displayed on the image-display LCD 1 4 via the digital/analog conversion cir- 
cuit 1 3 and Is applied also to a memory card 30. 

30 [0048] Rg. 2 illustrates a file (a synthesis target image memory file) that has been stored in the synthesis target 
image memory 6. 

[0049] The synthesis target image memory file includes a tag area and an Image data storage area. 

[0050] Synthesis target Image data representing a plurality of synthesis target images obtained by photography 

under different photographic conditions has been stored In the image data storage area. More specifically, strobe-pho- 

35 tography synthesis target image data obtained by photographing a subject using a strobe flash, frontlit-photography 
synthesis target image data obtained by photographing a subject with f rontlighting, and cloudy-weather/backlit-photog- 
raphy synthesis target image data obtained by photographing a subject in cloudy weather have been stored. 
[0051 ] A Strobe-photography synthesis target image represented by the strobe-photography synthesis target image 
data Is illustrated in Fig. 3, a frontlit-photography synthesis target image represented by the frontlit-photography synthe- 

40 sis target image data is illustrated in Fig. 4, and a cioudy-weather/backlit-photography synthesis target image repre- 
sented by the cloudy-weather/backlit-photography synthesis target image data is illustrated in Rg. 5. 
[0052] In this example, photography in cloudy weather and photography with backlighting are taken as being pho- 
tography under identteal photographic conditions. However, photography in cloudy weather and photography with back- 
lighting may be treated as photography under separate photographic conditions, and image data representing images 

45 obtained under separate photographic conditions may be stored separately in the synthesis target image memory file. 
[0053] Photographic condition information of each item of image data that has been stored in the Image data stor- 
age area is stored In the tag area in a form linked to the corresponding item of image data (or together with linldng infor- 
mation). The photographic condition information is information representing photographic circumstances, namely 
whether image data is image data that has been obtained by photography with frontlighting or image data that has been 

50 obtained by cloudy-weather (or backlit) photography [information representing whether a photographic condition among 
photographic conditions is frontlighting or cloudy weather (backlighting) shall be refen-ed to as "photographic circum- 
stances infonnatlon"]; information representing image data is image data that has been obtained by strobe photogra- 
phy; information representing color temperature at the time of photography; and information representing average 
luminance at the time of photography. 

55 [0054] These items of photographic condition information are obtained at photography of a synthesis target image 
represented by image data that has been stored in the image data storage area. For example, photographic condition 
infomiation indicative of f rontiit photography or cloudy-weather photography can be obtained based upon the difference 
between a spot metering value and an average metering value prevailing at photography of the subject. Photographic 
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condition information indicating wfietlier pliotograpliy is strobe pliotograptiy or not can be obtained based upon wlietlier 
or not the strobe flashed at the time the subject was photographed. Photographic condition information representing 
color temperature can be obtained based upon the output of a color temperature sensor In a camera at the time the 
subject is photographed. Photographic condition information representing average luminance can be obtained based 
5 upon a metering value (obtained from Image data, which is input to the exposure control circuit 7, or from a metering 
element) prevailing at the time the subject is photographed. 

[0055] A synthesis target image also may be produced by manipulating image data (cutting out a specific portion 
thereof, such as a person) obtained by this Image sensing apparatus. 

[0056] Fig. 6 Illustrates a file (a background Image memory file) that has been stored in the background image 
10 memory 5. 

[0057] Background image data obtained by photography is stored in a background image memory file. In this 
embodiment, the background image memory file does not contain a tag area, unlike the synthesis target image memory 
file. Of course, the background image memory file also m^ be provided with a tag area in a manner similar to the syn- 
thesis target image memory file and photographic condition Information prevailing at the time the background Image is 

15 photographed may be stored there. In such case luminance Infbnmatlon obtained from the exposure control circuit 7, 
strobe flash information from the strobe flash circuit 9 and color temperature Infonnation output from the color temper- 
ature sensor 10 would also be applied to the background image memory 5 and stored there. 
[0058] Fig. 7 illustrates an example of a background Image represented by background image data obtained by 
photography and stored temporarily In the background image memory 5. This background Image is displayed on the 

20 Image-display liquid crystal display unit 14 before Image synthesis processing Is executed. 

[0059] Fig. 8 illustrates an example of a composite image obtained by combining a synthesis target image (the 
Image of a girl at the lower left) with the background image. A composite image of this kind is displayed on the image- 
display liquid crystal display unit 1 4. The composite image is one obtained after the position and size of the synthesis 
target image have been adjusted, in the manner described earlier, while observing the composite Image resulting from 

25 superposition. 

(2) Image synthesis processing 

[0060] Figs. 9 to 12 are flowcharts illustrating the procedure of image synthesis processing by the image sensing 

30 apparatus according to the first embodiment. 

[0061] Before image synthesis Is performed In the picture-taking and synthesis mode, strobe-photography synthe- 
sis target Image data obtained by strobe photography, f rontlit-photography synthesis target Image data obtained pho- 
tography with frontlighting, and cloudy-weather/backllt-photography synthesis target image data obtained by 
photography in cloudy weather are stored in the synthesis target Image memory 6, as described above. 

35 [0062] If the dial 21 is turned to establish the picture-taking and synthesis mode, the background image being 
formed on the image sensing device 2 is displayed on the Image-display liquid crystal display unit 14 at all times (as a 
moving picture) (step 71). If the composition of the background image has been decided, the user presses a shutter- 
release button (not shown). The background is photographed and background image data representing the background 
Image obtained by photography is stored temporarily in the background image memory 5. The background Image data 

40 representing the background image is read out of the background image memory 5 and is applied to the image-display 
liquid crystal display unit 14 via the Image processing circuit 11, synthesis processing circuit 12 and digital/analog con- 
version circuit 13, whereupon the background Image is displayed on the Image-display liquid crystal display unit 14 as 
a still picture. 

[0063] If the execute button 24 Is pressed (step 42), a transition is made to image synthesis processing. It Is deter- 
45 mined whether the background image was obtained by frontlit photography or backlit photography (step 43). This deter- 
mination is made by the exposure control circuit 7 based upon a difference between the luminance level of a 
predetermined area at the center of the background image and the average luminance level of the background image. 
Next, it is detemiined whether the background Image was shot using a strobe flash (step 44). This detennination is 
made based upon strobe infonnation output from the strobe unit 9. 
50 [0064] Next, one item of synthesis target image data is found by the synthesis target image search circuit 8 from 
the synthesis target image data representing a plurality of synthesis target images that have been stored in the synthe- 
sis target image memory 6. This search processing is as follows: Photographic condition infonnation is read out of the 
tag area of the synthesis target image file that has been stored In the synthesis target image memory 6. Synthesis tar- 
get image data for which the frontilghting/backilghting photographic drcumstances and strobe information contained in 
55 the photographic condition information that have been read out match those of the background image Is read out of the 
synthesis target image memory 6 ("YES" at step 45). 

[0065] If synthesis target image data possessing a tag area which matches the result of the detennination regard- 
ing the f rontlighting/backlighting photographk: circumstances of the background Image and the result of the detemiina- 
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tion regarding the strobe has not been stored in the synthesis target Image memory 6 ("NO" at step 45), then strobe- 
photography synthesis target image data is selected from the image data that has been stored in the synthesis target 
Image memory 6 (step 46). The strobe-photography synthesis target Image data that has been selected is read out of 
the synthesis target image memory 6. 

5 [0066] Furthermore, the background-image color temperature sensed by the color temperature sensor 1 0 is read 
out of a memory (not shown) in the image processing circuit 11 (step 47). It is determined whether the difference 
between the color temperature of the synthesis target Image read out of the synthesis target image memory 6 and the 
color temperature of the background image just read out falls within an allowable range (step 48). 
[0067] If the difference is not within the allowable range ("NO" at step 48), then first color correction processing 

10 regarding the synthesis target image is executed (step 50). First color correction processing will be described later. 
[0068] If the difference between the color temperature of the synthesis target image read out of the synthesis target 
image memory 6 and the read color temperature of the background image falls within the allowable range ("YES" at 
step 48), then the processing of step 50 Is skipped. 

[0069] Next, luminance Information concerning the background image is read out (step 51). The luminance infbr- 
15 mation is obtained based upon the badcground image data applied to the exposure control circuit 7. It Is determined 

whetherthe luminance difference between the average luminance of the synthesis target image read out of the synthe- 
sis target image memory 6 and the read average luminance of the background image falls within predetermined values 
(step 52). 

[0070] The details are as follows: 
20 [0071] Rrst, the Image data of each pixel of RGB In the background image Is converted to a luminance value Ey of 
each pixel In accordance with Equation 1. 

Ey = 0.21 25R + 0.71 54G + 0.0721 B Eq. 1 

25 [0072] Average luminance data Is calculated from the luminance data of each pixel obtained by the conversion. Let- 
ting AEyl (which has been stored In the tag area of the synthesis target image memory file) represent the average lumi- 
nance regarding the synthesis target Image and letting AEy2 represent the average luminance regarding the 
background image, we calculate an absolute value AEyS of the average luminance difference of each Image In accord- 
ance with Equation 2. 

30 

AEy3 = IAEy1 -AEy2l Eq.2 

[0073] It is determined whetherthe absolute value AEy3 of the average luminance difference calculated in accord- 
ance with Equation 2 is equal to or less than a predetermined value (P, where P preferably is eight levels among 256 
35 levels of 0 to 255) (if Equation 3 is satisfied, then it is judged that the difference falls within predetermined values). 

AEy3 < P Eq. 3 

[0074] If the luminance difference does not fall within the predetermined values ("NO" at step 52), second color cor- 
40 rection processing concerning the synthesis target image is executed (step 54). The details of second color oon^ction 
will be described later. 

[(X)75] The synthesis processing circuit 1 5 executes synthesis processing for superimposing the background Image 
data and synthesis target image data (adds the data of the two images) (step 55). The data obtained by superposition 
is applied to the image-display liquid crystal display unit 14 via the digital/analog conversion circuit 13. The synthesis 
4S target image is displayed superimposed on the background image on the image-display liquid crystal display unit 14 

(step 72). 

[0076] In a case where it is desired to combine a synthesis target image different from the synthesis target image 
being displayed on the display screen of the image-display liquid crystal display unit 14, the use selects another syn- 
thesis target image ("NO" at step 56; step 57). The dial 21 is turned to establish the synthesis target Image selection 

50 mode. By pressing the push switch 23, synthesis target images represented by synthesis target image data that has 
been stored in the synthesis target image memory 6 are displayed on the image-display liquid crystal display unit 14 In 
order one frame at a time. When the desired synthesis target image is displayed, the user presses the execute button 
24 to thereby select the synthesis target image. In a case where a new synthesis target Image has been selected, 
processing from step 47 onward is repeated. 

55 [0077] The user observes the synthesis target image being displayed on the display screen of the image-display 
liquid crystal display unit 14 and, upon deciding that this synthesis target image should be combined with the back- 
ground image, presses the execute button 24 ("YES" at step 56). 

[0078] Next, the user observes the synthesis target image (which has been superimposed on the background 
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image) being displayed on the display screen of the image-display liquid crystal display unit 1 4 and determines whether 
the color balance of the synthesis target image is acceptable (step 58). If the color balance Is acceptable, the user 
presses the execute button 24. 

[0079] Fig. 1 3 illustrates an example of a screen for manual setting of color balance displayed on the manual-dis- 
5 play liquid crystal display unit 25. An arrangement may be adopted In which the manual color-balance setting screen is 
displayed in response to a command provided by the user. 

[0080] The manual color-balance setting screen includes an area 81 set by the user when the color red is adjusted, 
an area 85 set by the user when the color green is adjusted, and an area 89 set by the user when the color blue is 

adjusted. 

10 [0081 ] The red-color area 81 includes an area 82 touched by the user when the amount of red component is to be 
increased, an area 83 touched by the user when the amount of red component is to be decreased, and a setting display 
area 84 for displaying how far the set red component is from a reference amount (the amount of red component when 
no color balance adjustment has been applied). The green-color area 85 includes an area 86 touched by the user when 
the amount of green component is to be increased, an area 87 touched by the user when the amount of green compo- 

15 nent is to be decreased, and a setting display area 88 for displaying how far the set green component is from a refer- 
ence amount (the amount of green component when no color balance adjustment has been applied). The blue-color 
area 89 includes an area 91 touched by the user when the amount of blue component is to be increased, an area 92 
touched by the user when the amount of blue component is to be decreased, and a setting display area 93 for displaying 
how far the set blue component is from a reference amount (the amount of blue component when no color balance 

20 adjustment has been applied). 

[0082] The user sets the color balance using the areas 81 to 93. 

[0083] Figs. 14A, 14B, 14C illustrate the relationship between color-balance setting values (which con-espond to 
the setting values being displayed in the setting display areas 84, 88 and 89) and the correction values. The color bal- 
ance correction is can'ied out based upon these relationships. As will be described later, Ka represents a correction 
25 quantity regarding color data a* In an L*a*b* colorlmetric system, and Kb represents a correction quantity regarding 
color data b* in the L*a*b* colorimetric system. 

[0084] If color balance is set by the user (step 59), color-temperature identification data used in third color correc- 
tion processing, described later, is set to CB1 (step 60). (Color-correction identification data takes on a value of CBO or 
CB1 . The data Is set to CBO when the power supply is turned from oft to on.) Thereafter, third color correction process- 
so Ing regarding the synthesis target image is executed (step 61). The details of third color correction processing will be 
described later. 

[0085] If it is unnecessary to adjust the color balance of the synthesis target image being displayed on the Image- 
display liquid crystal display unit 14 ('YES' at step 58), the user presses the execute button 24 and processing sidps 

from step 59 to step 61 . 

35 [0086] Next, by observing the synthesis target Image (which has been superimposed on the bacl<ground screen) 
being displayed on the image-display liquid crystal display unit 14, the usercheclcs the brightness of the synthesis tar- 
get image (step 62). If the brightness is acceptable ('YES' at step 62), the user presses the execute button 24. 
[0087] In a case where the brightness of the synthesis target image being displayed is unacceptable and the user 
adjusts the brightness ('NO' at step 62), a manual setting area 94 shown in Fig. 15 being displayed on the manual-dis- 

40 play liquid crystal display unit 25 is utilized. An arrangement may be adopted in which the manual setting area 94 also 
is displayed in response to a display command provided by the user. 

[0088] The manual setting area 94 includes an area 95 touched by the user when the brightness of the synthesis 
target Image Is to be increased; an area 96 touched by the user when the brightness of the synthesis target Image is to 
be decreased; and a setting display area 97 for displaying how far the brightness is from a brightness reference amount 

45 (which indicates brightness when no brightness adjustment has been applied). 

[0089] If brightness is set manually by the user, luminance identification data used in second color correction 
processing, described later, is set to LM1 (step 64). (Luminance identification data tal^es on a value of LMO or LM1 . The 
data is set to LMO when the power supply is turned from off to on.) On the basis of the set brightness (the value being 
displayed in the setting displ^ area 97 con^sponds to a brightness setting value in Fig. 16), a con'ection quantity Kgy 

50 shown in Fig. 16 is calculated and brightness of the synthesis target image is corrected by second color correction 
processing (step 65, this second color con'ection processing being processing identical with that of step 54). 
[0090] Next, the position of the synthesis target image (superimposed on the bacl<ground image) being displayed 
on the image-display liquid crystal display unit 1 4 is adjusted by manipulating the arrow key pad 22. Further, the size of 
the synthesis target image is adjusted by pressing the push switch 23 (step 66). If the position and size of the synthesis 

55 target image have been decided, the user presses the execute button 24 so that the synthesis target image is combined 
with the background image (step 68). (The synthesis processing circuit 1 5 cuts the portion of the synthesis target image 
from the background Image and fits the synthesis target image data into the cut-out portion to generate composite 
Image data.) The result is displayed on the image-display liquid crystal display unit 14. 
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[0091 ] The composite image data representing the composite image obtained by synthesis processing is stored on 
the memory card 30 (step 69). 

[0092] Fig. 1 7 is a flowchart illustrating the procedure of first color correction processing. The first color correction 

processing subjects the synthesis target image to a color (white) balance correction. 
5 [0093] In first color correction processing, color temperature information of the synthesis target image is first read 
out of the tag area of the synthesis target image memory file (step 101). 

[0094] TTie image data representing the synthesis target image is 8-bit image data for each of R, G, B. This 8-bit 
image data R, G, B is converted to Image data of tristimuius values XYZ (steps 1 02, 1 03). The image data of the tris- 
timuius values XYZ obtained by the conversion undergoes a color balance correction. 
10 [0095] More specifically, the 8-bit image data R, G, B is converted temporarily to non-linear RGB data Rq, Gq, Bq. 
The non-linear RGB data Rq, Gq, Bq is further converted to linear RGB data Ri, Gi, Bi. Rnally, the linear RGB data R^, 
Gi, Bi is converted to tristimuius values XYZ. 

[0096] In general, relations defined by Equations 4 hold between the 8-bit image data R, G, B and non-linear RGB 
data Ro. Gq, Bq. The 8-bit Image data R, G, B is converted to the non-linear RGB data Rq, Gq, Bq in accordance with 
15 Equations 4. 

Ro = R/255 Eq.4 
Go = G/255 



20 



40 



Bo = B/255 



[0097] If consideration is given to a characteristic (decided by a function f), which is specific to the image sensing 
apparatus, between the non-linear RGB data Rq, Gq, Bq and linear RGB data Rf , G-i, B^, a relation indicated by Equa- 
ls tion 5 is established, where V g = R g , Gq, Bq, V = R ^ , G-i , B^ hold. 

Vo = f(V) Eq.5 

[0098] The non-linear RGB data Rq, Gq, Bq is converted to the linear RGB data R^, G^, B^ using Equation S. For 
30 example, if the y characteristic of the Image sensing apparatus is in compliance with ITU-RBT.709 (INTERNATIONAL 
TELECOMMUNICATION UNION ITU Radiocommunlcation BT Series 709), then Equation 5 can be expressed by 
Equation 6. The non-linear RGB data Rg, Gg, Bg is converted to the linear RGB data R^, G^, B.| using the relation of 
Equation 6. 



Vo = 



1.099 X V°-*' - 0.099 0.018s V sl.O 

4.50 X V 0.0s V <0.018 

... Eg. 6 



45 

[0099] The conversion from the linear RGB data Ri, Gi, B-i to the tristimuius values XYZ is perfonned in accord- 
ance with Equation 7. 



X/100' 




^1 


Oi2 


^3 






y/100 












... EC[. 7 


z/ioo. 






«32 


^3. 


A. 





55 

The matrix of a-ii ~ 833 used in Equation 7 is calculated based upon the characteristics of a color filter in the image 
sensing device 2 and the color temperature of the subject (step 102). The linear RGB data R^, G^, B^ is converted to 
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the tristimulus values XYZ in accordance with Equation 7 using the matrix calculated (step 103). The 8-bit image data 
R, G, B is thus converted to the tristimulus values by the foregoing processing. 

[0100] The color temperature of the bacl<ground image is read out of the memory in image processing circuit 1 1 
(step 105). 

5 [0101] Con-ection coefficients used in correction of color balance are calculated from the color temperature of the 
synthesis target image and the color temperature of the background image (step 106). The processing for calculating 
the correction coefficients is canied out In the manner described below. 

[01 02] In general, color temperature T and x [x = X/(X+Y+Z) , X, Y, Z are tristimulus values) are related as indicated 
by Equations 8. 

10 

X = -4.6070 X 1 0 + 2.9678 x 1 0 V ^ + 0.09911 x 1 0 V + 0.244063 (4000K< T <7000K) Eq. 8 

X = -2.0064 X 10^/7^ + 1.9081 X 10^/T^ +0.24748 x 10 ^/T + 0.237040 (7000KS T <25000K) 

15 Further, y [y = Y/(X+Y+Z) , X, Y, Z are tristimulus values) and x are related as indication by Equation 9. 

y = -3.000X ^ + 2.870X - 0.275 Eq. 9 

[0103] The values of x and y are calculated from Equations 8 and 9. The calculated x and y are substituted into the 
20 above-cited relations x = X/(X+Y+Z) , y = Y/(X+Y+Z) . If Y = 1 00.0 (= Yn) holds, then X and Z are calculated. The X, Y, 
Z thus obtained are white points X^, Y^, Zp. 

[0104] White points Xni, Ynii Zpi of the synthesis target image at the time of photography are calculated using the 
color temperature of the synthesis target image, and white points Xn2, Yn2, of the background image at the time of 
photography of the background image are calculated using the color temperature of the background Image. 
25 [0105] Conxion coeffidents Xna/Xp^ , Y^g/Yni , Znz/Zni are calculated using the cateulated white points X„] , Ypi , 
Zni, Xpg, Yn2. 2n2. In accordance with Equation 10, the calculated con-ection coefficients are multiplied by the tristimu- 
lus values X, Y, Z of the synthesis target image to thereby carry out the color balance correction [where X-g, Y^, Z^ are 
tristimulus values after the color balance connection (i.e., data representing the synthesis target image after the color 
balance con-ection)] (step 108). 

30 

X,=X^/X„ixX Eq.10 
Yi=Y„2/Y„iXY 

35 Zi=Z^/Z„iXZ 

[0106] Of course, a Von Kries color adaptation prediction equation may be used instead of Equation 10. 

[0107] The tristimulus values {X^y^.Z^) that have undergone the color balance correction are converted to RGB 

data (step 109). 

40 [0108] Fig. 1 8 is a flowchart illustrating the procedure of second color con-ection processing, and Fig. 1 9 illustrates 
the relationship between photographic circumstances such as frontlighting and backlighting and con^ion values. 
[0109] The second color correction processing subjects the synthesis target image to a brightness correction. 
[0110] The brightness correction In second color correction processing includes an automatic brightness correction 
(step 54 in Rg. 10) performed In a case where the luminance difference between average luminance of the synthesis 

45 target Image and average luminance of the background image does notfall within predetermined values, and a manual 
brightness correction (step 65 in Fig. 12) in which brightness con-ection values are set manually by the user. 
[0111] In second color con^ction processing, first it is determined whether the luminance identification data has 
been set to LMO or not in order to judge whetiier the automatic brightness correction or the manual brightness correc- 
tion is to be carried out (step 111). 

so [0112] If the luminance identification data has been set to LMO ("YES" at step 111), then, in order to perform the 
automatic brightness correction, luminance information concerning the background image is read (step 112) in a man- 
ner similar to that of the reading of the luminance information of the background Image described above (step 51 in Fig. 
10). 

[0113] Next, a luminance correction value is calculated from the luminance infomiation of the synthesis target 

55 image and the luminance Infomiation of the background image (step 113). More specifically, from the relationship 
shown In Fig. 19, first a luminance correction coefficient Ks is decided based upon the photographic circumstances as 
to whether the background image is frontlit or backlit (whether the image was captured indoors or at night is determined 
based upon the color temperature). As indk^ted by Equation 1 1 , the luminance con-eidlon value K^y is calculated by 
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multiplying the determined luminance correction coefficient Ks by the luminance difference between calculated average 
luminance AEy1 of the synthesis target image and average luminance AEy2 of the background image. 



KEY = Ks(AEy1 -AEy2) Eq. 11 

[0114] A brightness correction is performed next (step 115). 

[0115] If the correction is the automatic brightness correction, the calculated brightness con^ction value K^y is 
added to each Item of the RGB synthesis target Image data R, Q, B, as Indicated by Equations 1 2. As a result, synthesis 
target Image data having brightness-corrected R^ data, G2 data and B2 data is obtained. 

R2 = R + Key Eq. 12 



IS 



62 = 6 + KgY 
B2 — B + 



[0116] If the luminance identification data has not been set to LMO ("NO" at step 1 11), it is judged that the correction 
is the automatic brightness correction and the brightness correction value Kgy (see Rg. 16) that was set at step 63 in 
Fig. 12 is read (step 114). The brightness con-ection is carried out by adding the read brightness con^ction value 
20 to the RGB synthesis target image data. 

[0117] In Fig. 19, the correction coefficient Ks for backlighting differs from that for cloudy weather. However, an 
average value of the correction coefficient for bacl<llghting and the correction coefficient for cloucfy weather may be used 
as a common correction coefficient for both backlighting and cloudy weather. 

[0118] Rgs. 20 and 21 are flowcharts Illustrating the procedure of third color con-ection processing regarding the 
2s synthesis target image data. 

[0119] First, the color temperature information of the synthesis target Image Is readoutof the tag area of the syn- 
thesis target image memory 6 (step 1 60). 

[0120] In third color correction processing, a matrix for converting the RGB Image data to the tristimulus values XYZ 
is calculated in the manner described above (step 1 02). The RGB image data is converted to the tristimulus values XYZ 
30 using the calculated matrix. Furthennore, the tristimulus values obtained by the conversion are converted to Image data 
In the L*a*b* colorimetric system (step 162). The tristimulus values XYZ and the L*a*b* colorimetric data obtained by 
the conversions are related as indicated by Equations 13 and 14. The image data of the L*a*b colorimetric system is 
obtained using Equations 13 and 14. 



35 L*=1iexg(Y/Y„)-16 Eq. 13 

a* = 500x[g(Xfl(„)-g(Y/Y„)] 
b* = 200xtg(Y/Y„)-g(Z/Z„)] 

40 



g(P/P n) = (P/P „) ^'^ P/P „ > 0.008856 Eq. 1 4 



« g(P/P „) = 7.787 X (P/P n) + 1 6/11 6 P/P n S 0.008856 



P = X,Y,Z 



SO 

[0121] It is determined Whether the color-temperature identification data has been set to CB1 (step 172) 
[0122] If the color-temperature identification data has been set to C61 ('YES' at step 1 72), a color-balance con-ec- 
tion value that has been set manually is read (step 175). 

[0123] The color data a*, b* is extracted from the image data of the synthesis target image in the L*a*b* colorimetric 
55 system (step 164). 

[0124] Color balance correction processing is applied to the color data a*b* in the L*a*b* colorimetric system of the 
synthesis target image using the manual color-balance con-ection value that has been read (step 176). More specifi- 
cally, a con-ection quantity is read with reference to Figs. 14A to 14C in conformity with the setting value set manually 
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by the user. The read correction quantity Ka is added to the color data a*. Further the read correction quantity Kb is 
added to the color data b*. 

[0125] If the color balance correction is completed, the image data of the L*a*b* colorimetric system is converted 

to RGB image data (step 1 77). 

5 [01 26] If the color-balance identification data has not been set to CB1 ("NO" at step 1 72), a color balance correction 
Is not carried out (the processing of steps 175 and 176 is skipped). 

Second Embodiment 

10 [01 27] Figs. 1 to 8 and Rgs. 1 3 to 1 9 in the first embodiment can be applied to the second embodiment as Is. 
[0128] In the first embodiment described above, synthesis target image data representing synthesis target Images 
shot under a plurality of different photographic conditions is stored in the synthesis target image memory 6 and a single 
synthesis target image Is selected from the plurality of synthesis target Images. In the second embodiment, on the other 
hand, synthesis taiget image data representing a synthesis target image shot under a predetermined photographic 

15 condition is stored in the synthesis target image memory 6. The synthesis target image data representing this single 
synthesis target image is subjected to a color correction and combined with the background image. 
[0129] Figs. 22 to 25 are flowcharts illustrating the procedure of synthesis processing. Processes in these Figures 
identical with those shown in Rgs 9 to 12 are designated by like reference characters and a description thereof is omit- 
ted. 

20 [0130] It is determined whether information indicating whether a synthesis target Image represented by synthesis 

target image data read from the synthesis target image memory 6 is frontlit or backlit matches information Indicating 
whether a background image is frontlit or backilt and whether strobe information concerning the synthesis target image 
matches strobe information concerning the background image (step 46). 

[0131] If all of these items of information match ("YES' at step 45), contrast identification data is set to CNO (step 
25 121). (The contrast identification data takes on a value CNO, CN1 or CN2. The data is set to CNO as an initial value 
when the power supply Is turned from off to on.) If at least one of these items of information does not match (*N0* at 
step 45), then the color-temperature identification data is set to CBO and the contrast identification data is set to CN2 
(step 122), whereupon fourth color connection processing is executed (step 123). 

[0132] Further, it is determined whether the difference between the color temperature information of the synthesis 
30 target image and the color temperature information of the background image falls within an allowable range (step 48). 
[0133] If the difference between these items of color temperature infomnation does not fall within the allowable 
range ("NO" at step 48), then the above-described first color connection processing is executed (step 124). If the differ- 
ence between these items of color temperature Information does fall within the allowable range ("YES* at step 48), then 
the processing of step 124 is skipped. 
35 [0134] Further, if the absolute value of the luminance difference between the average luminance of the synthesis 
target image and the average luminance of the background image does not fall within predetermined values ("NO" at 
step 52), then the luminance identification data is set to LMO (step 53), as described earlier. Color correction processing 
Is subsequently applied to the synthesis target image as set forth earlier (step 54). 

[0135] A manual connection of contrast can be earned out as well. If the contrast of the synthesis target image (the 
40 luminance ratio of the darkest portion of the synthesis target image to the brightest portion thereof) differs (very greatly) 
from the contrast of the background image (the luminance ratio of the darkest portion of the background Image to the 
brightest portion thereof) ("NO" at step 127), then a contrast correction value regarding the synthesis target image is 
set manually (step 1 28). The manual-setting liquid crystal display unit 25 displays a manual-contrast setting screen 1 80 
shown in Fig. 26. (Naturally, an arrangement may be adopted in which the manual-contrast setting screen 180 is dis- 
45 played in response to a command from the user.) The manual-contrast setting screen 180 includes an area 181 touched 
by the user to raise contrast, an area 1 82 touched by the user to lower contrast and a setting display area 1 83 in which 
a contrast setting value is displayed. 

[0136] Fig. 27 is a graph illustrating contrast connection curves. 

[0137] A correction curve conforming to the contrast setting value (the value being displayed in the setting display 
50 area 1 83 of the manual-contrast setting screen 1 80 of Fig. 26) is selected and the con^ction is carried out in accord- 
ance with the selected correction curve. 

[0138] For example, if the set value of contrast in the manual-contrast setting screen 1 80 is '+2", the synthesis tar- 
get image is subjected to the contrast correction using the +2 curve. 

[0139] In a case where contrast correction value is set manually, the color-temperature identification data is set to 
55 CBO and the contrast identification data is set to CN1 (step 1 29). Thereafter, fourth con-ection processing is applied to 
the synthesis target image (step 130). 

[0140] When the color balance is set manually in the second embodiment ("NO" at step 58; step 59), the color-tem- 
perature identification data is set to CB1 and the contrast identification infonnatjon is set to CNO (step 60A). Fourth 
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color-correction processing the same as that of step 130 is subsequently applied to the synthesis target image (step 
61 A). 

[0141] Rgs. 28 and 29 are flowcharts Illustrating the procedure of fourth color correction processing. Fourth color 
con-ection processing corresponds to the third color correction processing and processing identical with that In the third 

5 color correction processing is designated by like reference characters and a description thereof is omitted. 

[0142] It is detennined whether contrast identification data is CNO (step 1 60). If the contrast identification data has 
been set to CNO ("YES" at step 1 61 ), the photographic circumstances as to whether the synthesis target image is frontlit 
or backlit and the strobe information of the synthesis target image match the photographic circumstances as to whether 
the bacl<ground image is frontlit or backlit and the strobe infbmiation of the background image, respectively. 

10 [0143] In this case, the luminance of the brightest portion of the synthesis target image and the luminEince of the 
brightest portion of the background image are approximately equal. Further, the luminance of the darkest portion of the 
synthesis target image and the luminance of the darkest portion of the background image are approximately equal. 
Accordingly, the contrast of the synthesis target image representing the luminance ratio of the brightest portion to the 
darkest portion of the synthesis target image and the contrast of the background image representing the luminance 

IS ratio of the brightest portion to the darkest portion of the background image are substantially equal. A conection of con- 
trast, therefore, is considered to be unnecessary and no contrast correction is set (step 170). 
[0144] If the contrast identification data is not CNO ("NO" at step 161), then it is determined whether the contrast 
identification data has been set to CN2 (step 165). 

[0145] If the contrast identification data has not been set to CN2 ("NO* at step 1 65), ('NO' at step 1 65), then a man- 
20 ual contrast con^ion value is read in such a manner that the contrast correction will be canled out based upon a man- 
ual contrast con-ection value (step 1 69). 

[0146] If the contrast identification data has been set to CN2 ("YES" at step 1 65), then the determinations concern- 
ing frontllghtingAjacklighting and use of strobe are performed with regard to the background Image in order to carry out 
an automatic contrast correction (steps 1 66, 1 67). A contrast correction value is calculated based upon the detennlna- 

2S tions concerning f rDntlighting/backlighting and strobe (step 1 68). For example, if the synthesis target image is an image 
obtained by strobe photography and the background image is an image obtained in sunny weather, a contrast con^c- 
tion value is calculated so as to obtain a contrast connection curve (e.g., a '+1 ' correction curve) that will assign a con- 
trast difference to the synthesis target image. If the synthesis target image is an image obtained in cloudy weather and 
the background Image is an Image obtained in sunny weather, a contrast correction value Is calculated so as to obtain 

30 a contrast correction curve (e.g., a '4-2* correction curve) that will assign a further contrast difference to the synthesis 
target image. 

[0147] RGB data representing the synthesis target image is converted to image data in the L*a*b* colorimetric sys- 
tem in a manner similar to that of the first embodiment (steps 102, 162). Brightness data L* and color data a*, b* are 
extracted from image data in the L*a*b* colorimetric system (steps 163, 164). 
35 [0148] If a contrast correction value Is calculated or a manual contrast correction value Is read, these values are 
used to perfomi a contrast correction with regard to the brightness data L* of the synthesis target image based upon 
the contrast connection curve (see Rg. 27) (step 171). It goes without saying that a color balance con-ection is carried 
out with regard to the color data a* and b* of the synthesis target image in a manner similar to that of the first embodi- 
ment (steps 172, 175, 176). 

40 [0149] In the foregoing embodiments, a case in which a picture-taking and synthesis mode has been set Is 
described. IHowever, the image sensing apparatus according to these embodiments can be set to other modes, e.g., 
the picture-taking mode and synthesis mode, as well. 

[0150] When the picture-taking mode has been set by the dial 21 , a subject is photographed by the image sensing 
device 2 and is converted to digital image data in the analog/digital conversion circuit 3. The digital image data is sub- 

4S jected to a gamma con-ection in the signal processing circuit 4. The image data passes through the background image 
memory 5 and Is applied to the digital/analog conversion circuit 13 via the image processing circuit 1 1 and the synthesis 
processing circuit 1 5. The image data is converted by the digital/analog conversion circuit 1 2 to an analog video signal, 
which is then applied to the image-display liquid crystal display unit 14. The image of the subject is displayed on the 
image-display liquid crystal display unit 1 4. The user decides the composition of the image while the user is viewing the 

50 image of the subject which is displayed on the image-display liquid crystal display unit 1 4. If the shutter-release button 
Is pressed, the image data representing the image of the subject is stored in the background image memory 5 tempo- 
rarily. 

[01 51 ] The image data is read out of the background image memory 5 and applied to the image processing circuit 
1 1 . The image processing circuit 1 1 subjects the image data to a color balance adjustment based upon color tempera- 
55 ture information providec^ by the color temperature sensor 1 0. If necessary, moreover, the image processing circuit 1 1 
executes brightness con-ection and other image processing. The image data merely passes through the synthesis 
processing circuit 15 and is applied to the memory card 30, where the data is recorded. 

[0152] When the synthesis mode is set by the dial 21, a subject to serve as background will have been photo- 
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graphed in the picture-taking mode and the data representing the background image will have been stored in the back- 
ground image memory 5. If the synthesis mode is set, the background image data is read out of the background image 
memory 5 and the background image is displayed on the image-display liquid crystal display unit 1 4. The synthesis tar- 
get image data is read out of the synthesis target Image memory 6 and Is displayed on the image-display liquid crystal 
s display unit 1 4 upon being superimposed on the background Image. The size and position of the synthesis target image 
are adjusted as set forth earlier. If the size and position are decided and the execute button 24 is pressed, the synthesis 
target image is combined with the background image to obtain a composite image in the manner set forth above. The 
data representing the composite image is applied to the memory card 30 and Is recorded thereon. 

w Claims 

1. An image sensing apparatus comprising: 

synthesis target image data storage means for storing data representing a plurality of synthesis target images 

IS obtained by photography under different photographic conditions; 

image sensing means for photographing background and outputting data representing a background image; 
synthesis target image selection means for selecting one item of synthesis target image data suited to the 
background image from a plurality of items of synthesis target image data, which have been stored in said syn- 
thesis target image data storage means, based upon photographic conditions prevailing at time of photography 

2(7 of the background Image by said image sensing means; and 

image synthesis means for combining the synthesis target Image selected by said synthesis target image 
selection means with the background image output from said image sensing means, and outputting composite 
image data representing a composite Image. 

25 2. An Image sensing apparatus according to claim 1 , wherein said synthesis target image selection means automat- 
ically selects one item of synthesis target image data suited to the background Image from the plurality of items of 
synthesis target image data based upon photographic conditions of the synthesis target image and of the back- 
ground image. 

30 3. An image sensing apparatus according to claim 1 , further comprising selection command input means for inputting 
a selection command; 

said synthesis target image selection means selecting the synthesis target image data from said plurality of 
items of synthesis target image data in dependence upon a selection command input from said selection com- 
35 mand Input means. 

4. An image sensing apparatus according to claim 1 , further comprising: 

photographic-condition storage means for storing photographic conditions prevailing at time of photography of 

40 the synthesis target image; and 

image adjustment means for applying image adjustment processing for at least one of a color adjustment and 
a luminance adjustment to a synthesis target image represented by the selected synthesis target image data 
based upon photographic conditions of the synthesis target Image corresponding to the synthesis target image 
data selected by said synthesis target image selection means and photographic conditions of the background 

45 image; 

said image synthesis means combining the synthesis target image, which has been subjected to image adjust- 
ment processing by said Image adjustment means, with the background Image. 

5. An Image sensing apparatus according to claim 1 , further comprising: 

50 

image adjustment command input means for inputting an image adjustment command for at least one of a 
color adjustment and luminance adjustment; and 

image adjustment means for applying image adjustment processing for at least one of a color adjustment and 
a luminance adjustment to a synthesis target Image represented by the selected synthesis target Image data 
55 based upon an image adjustment command input from said image adjustment command input means; 

said image synthesis means combining the synthesis target image data, which has been subjected to image 
adjustment processing by said Image adjustment means, with the background image data. 
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6. An image sensing apparatus according to claim 1 , further comprising: 

a display unit for displaying images; 

background image display control means for controlling said display unit so as to display the background image 
5 sensed by said image sensing means; and 

composite image display control means for controlling said display unit so as to display the composite image 
synthesized by said image synthesis means. 

7. An image sensing apparatus according to claim 6, further comprising: 

10 

adjustment command input means for inputting an adjustment command for at least one of position and size 
adjustment of a synthesis target image being displayed on said display unit; and 

adjustment means for applying an adjustment for at least one of position and size adjustment of the synthesis 
target image based upon the adjustment command input from said adjustment command input means. 

IS 

8. An image sensing apparatus comprising: 

synthesis target infonnation storage means for storing data representing a synthesis target image and photo- 
graphic conditions prevailing at time of photography of the synthesis target image; 
20 image sensing means for photographing background and outputting data representing a background image; 

and 

image adjustment processing means for applying image adjustment processing for at least one of a color 
adjustment and a luminance adjustment to a synthesis target image represented by synthesis target image 
data, which has been stored in said synthesis target information storage means, based upon the photographic 
25 conditions of the synthesis target image that have been stored in said synthesis target information storage 

means and photographic conditions of the background image sensed by said image sensing means; and 
Image synthesis means for combining the synthesis target image data that has been subjected to the Image 
adjustment processing by said image adjustment processing means with the background image data output 
from said image sensing means, and outputting composite image data representing a composite image. 

30 

9. An image sensing apparatus according to claim 8, wherein the synthesis target image data that has been stored in 
said synthesis target Image information storage means was obtained by strobe flash photography. 

10. An Image sensing apparatus according to claim 8, further comprising image adjustment command input means for 
35 Inputting an image adjustment command for at least one of a color adjustment and a luminance adjustment; 

wherein said image adjustment processing means applies image adjustment processing for at least one of 
a color adjustment and a luminance adjustment to a synthesis target image based upon an image adjustment com- 
mand input from said image adjustment command input means; and 

40 said image synthesis means combines the synthesis target image data, which has been subjected to image 

adjustment processing by said image adjustment means, with the background image data. 

11. An image sensing apparatus according to claim 8, further comprising: 

45 a display unit for displaying images; 

background image display control means for controlling said display unit so as to display the background image 
sensed by said image sensing means; and 

composite image display control means for controlling said display unit so as to display the composite Image 
synthesized by said image synthesis means. 

50 

12. An image sensing apparatus accorcling to claim 1 1 , further comprising: 

adjustment command input means for inputting an adjustment command for at least one of position and size 
adjustment of a synthesis target image being displayed on said display unit; and 
55 adjustment means for applying an adjustment for at least one of position and size adjustment of the synthesis 

target image based upon the adjustment command input from said adjustment command input means. 

13. An image sensing method comprising: 
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storing data representing a plurality of synthesis target images obtained by photography under different photo- 
graphic conditions; 

photographing background and outputting data representing a background Image; 
selecting one item of synthesis target image data suited to a background image from a stored plurality of items 
of synthesis target Image data based upon photographic conditions prevailing at time of photography of the 
background image; and 

combining the selected synthesis target image with background image data and outputting composite image 
data representing a composite Image. 

14. An Image sensing method comprising: 

storing data representing a synthesis target image and photographic conditions prevailing at time of photogra- 
phy of the synthesis target image; 

obtaining data representing a background image by photographing background; 

applying image adjustment processing for at least one of a color acljustment and a luminance adjustment to a 
synthesis target Image represented by synthesis target image data based upon the photographic conditions of 
the synthesis target image and photographic conditions prevailing at time of photography of the background; 
and 

combining the synthesis target image data that has been subjected to the Image adjustment processing and 
the background Image data, and outputting composite Image data representing a composite Image. 
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Fig. 2 
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Fig. 6 



BACKGROUND IMAGE 
MEMORY FILE 



BACKGROUND 
IMAGE DATA 



21 




22 




23 



EP 1 100 262 A1 



Fig. 9 



START ^ 


PICTURE 
ANDSYNTF 


-TAKING 
ESISMOOE 




DETERMINE 
PHOTOGRAPHIC 
CIRCUMSTANCES OF 
BACKGROUND IMAGE 



DETERMINE WHETHER 
STROBE USED FOR 
BACKGROUND IMAGE 



^4 



N 



SELECT STROBE- 
PHOTOGRAPHY 
SWHESIS TARGET 
IMAGE DATA 



DO 

PHOTOGRAPHIC 
CIRCUMSTANCES AND STROBE^ 
INFORMATION OF SYNTHESIS 
TARGET IMAGE MATCH 
PHOTOGRAPHIC QRCUMSTANCES 
.AND STROBE INFORMATION^ 
OF BACKGROUND^ 
JMAGE?^ 



46 



-45 



24 



EP 1 100 262 A1 



Fig. 10 
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Fig. 12 
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Fig. 15 
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Fig. 17 
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Fig. 20 
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Fig. 21 
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Fig. 22 
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Fig. 23 
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Fig. 24 
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Fig. 25 
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Fig. 26 



MANUAL COhJTRAST 



CONTRAST 












1 

+ 2 



-r 7 f- 

I8i 182 



Fig* 27 

AFTER CORRECTION 




EP 1 100 262 A1 



Fig. 28 
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Fig. 29 
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